When What You Hear is What You See: Presence and Auditory-Visual Integration in Virtual Environments

In this paper, it is hypothesized that consistency across modalities in terms of matching the visual space to the auditory space is important for the sense of presence. An experiment was carried out where thirty participants were exposed to four conditions having different degrees of auditory-visual consistency (one purely visual and three auditory-visual). A presence questionnaire was used after exposure to measure participants’ sensations. Although participants rated the auditory-visual conditions as inducing significantly higher presence than the condition with only visual information, no differences in presence ratings between the three auditory-visual conditions were found. However, participants’ rankings of their sensed presence in all conditions revealed that there might be such differences. Moreover, the results show that sound in general has a significant effect on VE users’ sense of presence. Keywords--Presence, auralization, auditory-visual integration, virtual acoustics, multimodal VE.